In search of requirements analyst characteristics that influence requirements elicitation effectiveness: a quasi-experiment

Context: Elicitation effectiveness depends on non-wellunderstood analyst?s skills and abilities. Identifying which analysts? characteristics have stronger influence on elicitation may help to improve requirements quality. Objective: Identify the analysts? characteristics that influence on the elicitation effectiveness. Method: We analyzed the impact of: the analyst?s experience in interviews, elicitation and requirements; their academic qualifications, the familiarity with problem domain and the time spent during the elicitation session in the effectiveness of the elicitation and subsequent consolidation of requirements, using a quasi-experiment. Results: The knowledge of the problem domain, the analysts? academic qualifications and the elicitation time do not appear to influence the effectiveness of the elicitation sessions. The analyst?s experience exerts a slight negative influence on the effectiveness of the elicitation session. The analyst?s experience and familiarity with problem domain adversely affect the consolidation process. Finally, the analyst?s academic qualifications have a strong positive impact(statistically significant) on the effectiveness of the consolidation process. Conclusions: Although the evidence is still scarce, it seems the analyst's confidence on his own experience may be harmful in some cases. Specific training in software requirements may yield much higher gains than non-specific analyst experience

Effect of domain knowledge on elicitation effectiveness: an internally replicated controlled experiment

Context. Requirements elicitation is a highly communicative activity in which human interactions play a critical role. A number of analyst characteristics or skills may influence elicitation process effectiveness. Aim. Study the influence of analyst problem domain knowledge on elicitation effectiveness. Method. We executed a controlled experiment with post-graduate students. The experimental task was to elicit requirements using open interview and consolidate the elicited information immediately afterwards. We used four different problem domains about which students had different levels of knowledge. Two tasks were used in the experiment, whereas the other two were used in an internal replication of the experiment; that is, we repeated the experiment with the same subjects but with different domains. Results. Analyst problem domain knowledge has a small but statistically significant effect on the effectiveness of the requirements elicitation activity. The interviewee has a big positive and significant influence, as does general training in requirements activities and interview experience. Conclusion. During early contacts with the customer, a key factor is the interviewee; however, training in tasks related to requirements elicitation and knowledge of the problem domain helps requirements analysts to be more effectiv

Evidence of the presence of bias in subjective metrics: analysis within a family of experiments

Context: Measurement is crucial and important to empirical software engineering. Although reliability and validity are two important properties warranting consideration in measurement processes, they may be influenced by random or systematic error (bias) depending on which metric is used. Aim: Check whether, the simple subjective metrics used in empirical software engineering studies are prone to bias. Method: Comparison of the reliability of a family of empirical studies on requirements elicitation that explore the same phenomenon using different design types and objective and subjective metrics. Results: The objectively measured variables (experience and knowledge) tend to achieve more reliable results, whereas subjective metrics using Likert scales (expertise and familiarity) tend to be influenced by systematic error or bias. Conclusions: Studies that predominantly use variables measured subjectively, like opinion polls or expert opinion acquisition

Proposal of a Quasi-Experiment for studying the effect of experience on elicitation effectiveness

We plan to perform a quasi experiment to evaluate the effect of experience on requirements elicitation. Researchers will play the role of customers, whereas participants will perform the role of analysts. Analysts will hold a 60 minute interview and will then be given 25 minutes to write up a report of their findings. Participant effectiveness will be compared with available data series on the effectiveness of novice analysts that we have collected previously